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ABSTRACT 

Traditional methods for task analysis have been 
largely based on the Instructional Systems Development (ISD) model, 
which is widely used throughout industry and the military. The firsst 
part of this document gives an overview of cognitive task analysis, 
which is conducted within the first phase of ISD. The following steps 
of cognitive task analysis aro outlined: (1) identification of key 
job tasks and training issues r (2) visual representation of the 
knowledge structure; (3) description of cognitive processes 
underlying performance; (4) identification of differences between 
experts and less-experienced personnel; and (5) implications of 
results for the design phase. Following this overview, the paper 
presents, in outline form, guidelines that describe the procedures 
and deliverables for a cognitive task analysis. The deliverables 
(research proposal, interim report, and final report) are described, 
and the information contained in them is outlined. Guidelines for 
conducting a cognitive task analysis are presented in the following 
steps: establishing analytic goals, selecting subjects, selecting 
data collection methods, analyzing data, validating results, 
communicating and interpreting results, and formulating training 
recommendations. Attachments include two figures that provide 
examples of network scaling and an example of an expert mental model. 
(YLB) 
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Abstract 

Traditional methods for task analysis have been based largely on the Instructional Systems 
Development (ISD) model, which is widely used throughout industry and the military. The ISD 
model includes five phases: (Task) Analyze, Design, Develop, Implement, and Control. 
Current ISD analysis methods have proven effective for analysis and training of psychomotor 
skills, but offer little insight into jobs involving complex cognitive skills, which increasingly 
require training today. 

Cognitive methods have been used to conduct task analyses to support training development, 
particularly where the job to be trained involves complex cognitive skills requiring decision- 
making, problem-solving, attention-switching, and/or the effective organization and retrieval of 
vast amounts of knowledge. Cognitive task analysis examines the cognitive processes and 
structures that support skilled work performance, and identifies important differences between 
experts as compared with those having less expertise. 

Cognitive methods need to be incorporated into ISD task analysis. To date, however, no 
standard procedure has been developed for planning or conducting a cognitive task analysis. 
This paper presents a standard operating procedure for conducting a cognitive task analysis 
within the existing ISD analysis framework. The step-by-step presentation describes planning 
and conducting cognitive task analysis as part of the ISD process. Steps involved in establishing 
analytic goals, selecting subjects, selecting data collection methods, analyzing data, validating 
results, communicating and interpreting results, and formulating training recommendations are 
outlined in laymen's terms. 
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COGNITIVE TASK ANALYSIS 

OVERVIEW: CONDUCTING COGNITIVE TASK ANALYSIS 

1. Purpose . This section provides an overview of cognitive task analysis conducted within the 
analysis phase of ISD. Cognitive task analysis determines the mental processes and skills 
required to perform a task at high proficiency levels and the changes that occur as the skills 
develop. Cognitive task analysis helps determine how a task is learned, how learning can 
be facilitated, and what cognitive processes underlie and support effective job performance. 
This information is then used in the design pha^ to determine the most effective manner 
of training employees to achieve high levels of proficiency in job performance. 

2. Critical Actions . During the cognitive task analysis, the following actions take place: 

• Identification of key job tasks and training issues 

Identify major analytic goals 
Identify training issues or problems 
Select tasks suitable for cognitive analysis 

• Visual representation of the knowlec'ge structure 

• Description of cognitive processes underlying performance 
. Skills 

Conceptual and procedural knowledge 
Learning and performance strategies 

• Identification of differences between experts and less-experienced personnel 

Differences in knowledge structures (Optional) 
Differences in skills 

Differences in conceptual and procedural knowledge 
Differences in learning and performance strategies 
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2. Critical Actions . (Continued) 

• Implications of results for the design phase 

Training structure 
Training sequence 

Selection of instructional setting, media, and strategies 

3. Documentation . The cognitive task analysis includes production of the following 
deliverables: 

• Research Proposal 

• Interim Report (Optional) 

• Final Cognitive Task Analysis Report 

These deliverables are described on page 7 (Guidelines for Conducting Cognitive Task 
Analysis). 

The research proposal explains how the specifications are to be met, by presenting a design 
for the research. The optional interim report details the progress made toward meeting tlie 
specifications, and describes any proposed modifications to the research proposal. The 
final report presents the complete results from the implementation of the research proposal. 

4. Discussion of Cognitive Task Analysis . Cognitive task analysis examines the cognitive 
structures and processes (e.g. , problem-solving, decision-making, attention allocation, self- 
monitoring, performance strategies, and knowledge organization) underiying expert 
performance. Comparisons are made between experts and less-experienced personnel to 
identify effective job performance skills, knowledge, and knowledge structures. By 
examining changes in knowledge structures and mental processes as the job skills develop, 
cognitive task analysis provides an understanding of how a task is learned and how to 
fu ilitate learning. 

5. Guidelines for Conducting Cognitive Task Ana lysis. These guidelines define the 
procedures for accomplishing a comprehensive cognitive task analysis and preparing the 
cognitive task analysis deliverables. 

5.1 Identification of Kev Job Tasks and Training Issues . The overall goals of the analysis 
are discussed. Job tasks or functions suitable for a cognitive task analysis are 
identified. Training issues or problems associated with the tasks are identified so they 
can be addressed in the analysis. 
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COGNITIVE TASK ANALYSIS 

5.2 Vkufll Renres entarion of the Knowledge Structure. The overall organization of 
experts' or good performers' knowledge about the task is presented in a visual format 
such as a network of linked concepts, a tree diagram, or a schema depicting 
interrelationships between knowledge. 

5.3 Description of Cognitive Processes Underlying Performance . The key skills (e.g., 
long-range planning, projecting aircraft routes), knowledge, and learning and 
performance strategies that support effective task performance are described. 

5.4 Identification of Differences Between Experts a nd Less-Experienced Personnel. 
Differences are described between experts and less-experienced personnel in their 
knowledge structures, skills, knowledge, and learning and performance strategies. 
The differences are discussed in terms of how individuals learn the task, how learning 
might be facilitated, and how expertise for the task develops over time. 

5.5 Implications of Results for the Design Phase . The Final Cognitive Task Analysis 
Report discusses designing instruction so as to best facilitate progression from one 
proficiency level to the next. Areas covered include training structure, training 
sequence, and the selection of instructional setting, instructional strategies, and media 
delivery systems. 

6. Discussion . Cognitive task analysis includes a wide variety of data collection and analysis 
methods, with each analyst modifying or combining metnods to best fit the task to be 
analyzed. Additionally, because of the resource-intensive nature of cognitive task analysis, 
it may not always be possible to accomplish all of the analytic tasks outlined in the 
guidelines. For example, it may be desirable to limit the analysis only to experts, 
identifying only expert-typical knowledge structures, skills, and strategies. Or, the analysis 
may have the goal of examining only one aspect of expertise or job competence, such as 
effective learning and performance strategies. The Research Proposal should specify which 
analytic tasks will be completed. 

7. Su ggested References . The following references are suggested for more detailed 
information about systematic approaches to conducting cognitive task analysis, particularly 
within an ISD framework: 

Beckschi, P. F., Lierman, B. C, Redding, R. E., & Ryder, J. M. (1991). Prwg^ltiral 
guide for integrating cognitive methods into ISD task analysis . Special Report, Air Force 
Training Command. 

Bonar, J., et al. (1986). Guide to Cognitive Task Analysis. Pittsburgh: Learning 
Research and Development Center, University of Pittsburgh. 

Branson, R. K., & Grow, G. (1987). Instructional Systems Development. In R. Gagne 
(Ed.), Instructional Technology: Foundations . Hillsdale, NJ: Erlbaum. 
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7. Su ggested References . (Continued) 

Hoffman, R. R. (1987, Summer). The problem of extracting the knowledge of experts 
from the perspective of experimental psychology. AI Magazine, pp. 53-67. 

Ryder, J. M., & Redding, R. E. (1991). Integrating cognitive task analysis into 
Instructional Systems Development QSPI . Submitted for publication. (Available from 
Human Technology, Inc.) 

Ryder, J. M., Redding, R. E, & Beckschi, P. F. (1987). Training development for 
complex cognitive tasks. In Proceedings of the Human Factors Society - 31st Annual 
Meeting (pp. 1261-1265). Santa Monica, CA: Human Factors Society. 
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GUIDELINES FOR CONDUCTING 
COGNITIVE TASK ANALYSIS 



1. PURPOSE 

1.1 These guidelines describe the procedures and the deliverables for a cognitive task 
analysis conducted within the analysis phase of the ISD process. Cognitive task 
analysis is a systematic process for studying the cognitive processes and structures that 
support work performance. The analysis determines the knowledge structures, 
knowledge, skills, and strategies needed for high-level job performance, and identifies 
important differences between experts as compared with those having less experience. 

1.2 The deliverables of a cognitive task analysis include: 

a. Research Proposal. The research proposal explains how the specifications are 
to be met, by presenting a design for the research. 

b. Interim Report (Optional). The interim report details the progress made towaid 
meeting the specifications, and describes any proposed modifications to the 
research proposal. Because this deliverable is optional, it may be desirable for 
the interim report to discuss only selected items included in the guidelines. For 
instance, the report might only discuss modifications of the data collection and 
subject selection procedures based upon information obtained from pilot testing. 

c. Final Cognitive Task Analysis Report. The final report presents the complete 
results from the implementation of the research proposal. 

1.3 The task analysis research proposal, interim report, and final report present the 
following information: 



a. 


Goals of the task analysis 


b. 


Selection of tasks and issues studied 


c. 


Subjects 


d. 


Data collection methods 


e. 


Data analysis methods 


f. 


Results validation methods 


g- 


Results of the task analysis 
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1.4 In presenting the results of the cognitive task analysis, the final report may include: 

a. Visual representation of the knowledge structure 

b. Description of cognitive processes underlying performance 

• Skills 

• Conceptual and procedural knowledge 

• Learning and performance strategies 

c. Identification of differences between experts and less-experienced personnel 

• Differences in knowledge structures (Optional) 

• Differences in skills 

Differences in conceptual and procedural knowledge 

• Differences in learning and performance strategies 

d. Implications of results for the design phase 

• Training structure 

• Training sequence 

• Selection of instructional setting, media, and strategies 

2. GUIDELINES FOR CONDUCTING A COGNITIVE TASK ANALYSIS 

2. 1 Goals of the Task Analysis . Identify the major goals of the analysis, describe the job 
task(s) and training problems that are the focus of the analysis, and provide 
justification for the selections. Justify the need for conducting a cognitive task 
analysis on the task(s) and problem(s) identified. 

a. Identify major goals. 
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2.1 Go als of the Task Analysis . (Continued) 

b. The analysis must focus on a set of questions, issues, or problems. Delineate the 
specific training problems, learning needs, or questions that the analysis 
addresses. This information may include the identification of: 

• Long-term knowledge to be retained by the target population 

• Time-sharing skills to be developed by the target population 

• Effective cognitive-optimizing strategies 

• Effective knowledge organization for the task 

• Effective learning and performance strategies 

• Methods for improving the transfer of skills 

• Methods for developing better problem-solving skills 

c. Provide evidence for the training problems discussed in Item b (above). 

d. Select job tasks or functions to be analyzed. 

e. Provide reasons for job task selections. Is the entire job being analyzed? If not, 
task components or functions should be carefully selected to include those that 
represent major training problems, are of critical importance to the job, involve 
a high degree of cognitive skills, and/or are particularly well-suited for cognitive 
task analysis techniques; for example: 

• Critical tasks (criticality, difficulty) 

• Job bottlenecks 

• Tasks on which operators make frequent errors 

• High workload tasks requiring attention-switching or division of attention 
between task components 

• Tasks involving high performance skills (i.e., requiring massive amounts 
of practice to obtain proficiency) 

• Tasks with a significant judgmental component involving problem-solving 
or decision-making 
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2.1 Goals of the Task Analysis. (Continued) 

e. Provide reasons for job task selections. (Continued) 

• Tasks requiring large amounts of knowledge to be assimilated during 
training 

• Tasks that experts have considerable difficulty verbalizing or demonstrating 
through overt actions 

• Complex tasks in which there may be considerable variability in how an 
individual performs the task, due to a number of cognitive strategies 
available 

f. Provide evidence for the task characteristics discussed in Item e (above). 

£. Include a description or review of what is already known about the nature of 
expertise in the job task, based on information obtained from: 

• Technical manuals 

• Operating procedures 

• Interviews with subject-matter experts 

• Observation 

• Previous research 

• Professional literature 

• Occupational surveys 

• Agency orders 

2.2 Subjects . Describe the subjects of the analysis, covering the following points: 

a. Ability or Experience Levels. Describe the range of ability or experience levels 
represented in the sample. Experts and novices or good and poor oerformers 
should be included. Ideally, novices, apprentices, sojourners, and several 
different types of experts (e.g., supervisors, best-rated experts, experts with less 
experience) would be included if differences between experts and novices are 
being investigated. 
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2.2 Subje cts. (Continued) 

b. Measurement and Selection. Are the subjects within each group truly 
representative of the group? Describe the definition of "expert." How is 
expertise for this task defined and measured? This description might specify: 

• Years of experience 

• Performance ratings by supervisors 

• Proficiency or achievement test scores 

• Peer nomination 

• Time to complete task(s) 

c. Sample Size. Is sample size for each subject group large enough to get a 
representative range of results and a representative picture of performance? 

2.3 Data Collection Methods . Describe the methods used to collect data, covering the 
following points: 

a. Selection of Methods. Describe which methods are used. Explain how each 
method selected addresses one or more of the issues/questions/training problems 
that are the focus of the analysis. Explain why the method is appropriate for the 
type ot job task(s). If a relatively small sample size is used, ensure that the 
method is powerful and sensitive enough to reveal differences even in small 
samples. 

Cognitive task analysis methods typically fall into one of four general categories: 

• Protocol Analysis. This method involves having subjects think aloud while 
performing a task or while describing task performance. The verbalizations 
are then used to infer the individual's cognitive processing for the task. 

• Psychological Scaling. This method involves having subjects make 
judgments through sorting, rating, or ranking task-relevant knowledge, or 
measuring subjects' recall, response times, output orders, and/or errors. 
The results are used to obtain proximity estimates from the subjects' 
judgments, rankings, sortings, or responses to stimuli in order to infer 
psychological structure (for example, how concepts relating to the task are 
organized in memory). 
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2.3 Data Collection Methods . (Continued) 

a. Selection of Methods. (Continued) 

• Performance Modeling. When using this method, simulations are 
constructed of a job or task, and a model of task performance is developed 
which is tested and refined by varying inputs, operator characteristics, or 
task characteristics to evaluate their effects upon simulated performance. 

• Observation of Job Performance, and Interviewing. These traditional task 
analysis techniques are also used in cognitive task analysis. One 
modification of traditional observation is to induce modified task behavior, 
and then observe performance. Traditional interview techniques are often 
modified in order to elicit information about reasoning processes. 

b. Reliability and Validity of Methods Selected. What is the reliability and validity 
of each method used? Discuss the selected methodology in terms of past 
research findings, and describe how validity and reliability are established in the 
current study: 

• Conduct pilot testing on each data collection method and measurement 
Instrument. 

• Obtain interceding reliability ratings. 

• Obtain cross-validation from subject-matter experts. 

• Revise procedures based on the above findings. 

c. Controlled Experimental Design. Discuss the adequacy of the overall 
experimental design to ensure there are effective controls in place for accurate 
measurements and interpretation of results. Ensure the methodology conforms 
to sound scientific practice. Consider: 

• Controlled conditions 

• Internal validity 

• Potential confounding variables 

• Potential extraneous influences 

• Demand characteristics 

• Experimenter effects 

E± 13 



COGNITIVE TASK ANALYSIS 



2.3 Data Collection Methods . (Continued) 

• Realism of stimuli, scenarios, and task simulations 

• Clear and unambiguous instructions to subjects 

• Random assignment of subj^ts to treatment conditions 

• Adequate data collection and recording equipment (e.g. , high-fidelity audio 
and video) 

d. Convergent Methods and Measures. Discuss the adequacy of convergent 
methods. Are differing techniques used so as to provide cross-validation of the 
results obtained from the various data collection methods? Is there a 
representative sampling of problem types and subtasks (e.g., typical problems, 
tough problems, time-constrained problems, difficult subtasks, easy subtasks) in 
order to get at different aspects of task performance? 

e. Measures and Experimental Treatments. Are the measures and comparisons 
well-defined? 

• What different experimental treatments (conditions) are present? For 
example: 

High-complexity versus low-complexity problems 
High-difficulty versus low-difficulty problems 
Complete versus partial information provided 
Unlimited time versus limited time provided 

• What group comparisons are made? For example: 

Experts versus intermediates versus novices 

• What measures are taken for use in the data analysis? For example: 

Number of times a strategy is used 

Number of times a particular knowledge category is used 

Mean similarity ratings 

Time to complete task 
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2.3 Data Collection Methods. (Continued) 

f. Subsequent Iterations and Refinements. Discuss provisions for additional data 
collection subsequent to the initial data collection. How is the information 
gained during the early stages of data collection and analysis used to guide and 
fme-tune future task selection and analysis goals, and future data collection and 
analysis? 

2.4 Data Analysis Methods . Piscuss how the data are analyzed, cov^-ring the following 
points: 

a. Coding. What coding scheme is used and what is the rationale for using it? 
How are reliability and validity established for the coding? 

b. Data Analysis. What analysis is performed? Statistical analysis? Subjective 
content analysis? Model-building exercises? Provide the rationale for the data 
analysis method used— consider how it is appropriate, as related to: 

• Data collection method used 

• Statistical or other properties of the data (e.g. , whether normal distribution, 
restricted range of data, subjects' rankings on a questionnaire, small sample 
size, etc.) 

• Questions/issues of concern 

• Type of output desired (e.g., whether a mental model, network diagram, 
sets of production rules, etc.) 

2.5 Results Validation . Provide validation for the results obtained and for any models 
flowing from those results. Consult with multiple subject-matter experts in order to 
obtain validation for the models developed, the results obtained, and the interpretation 
of subjects' responses. The following should be discussed or considered: 

a. Completeness. How complete are the models? What aspects of performance do 
they describe? 

b. Significance. How important and useful are the models for understanding the 
job? 

c. Generalizability. How representative are the results, and to what extent can the 
findings be generalized to other individuals, tasks, and situations? 
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2.5 Results Validation . (Continued) 

d. Reasonableness. Do the results and models seem reasonable, based upon the 
experts' own experience, other results obtained, and what is already known about 
the job? 

e. Predictability. Do the models allow one to predict performance once certain 
parameters are known? 

f. Limitations of the Data. Discuss the results in terms of what information is 
provided, what remains unknown, and what is indicated for future research. 
Discuss applicable caveats and cautions in interpreting and generalizing the 
findings. Discuss the overall reliability and validity of the findings and the range 
of application of the models. 

g. Subsequent Iterations/Refinements. Discuss whether there is a need for model 
refinement or results verification through further data collection, analysis, or 
validation efforts. 

2.6 Results . Discuss the results by providing an explicit model of the knowledge 
structure, skills, knowledge, and strategies underlying expertise for the task(s). The 
following should be discussed or considered: 

a. What do the results reveal overall about the nature of expertise in the task? 
Describe the results in terms of: 

• Effective mental models for the task 

• Types of skills required for competent performanc, 

• Effective cognitive-optimizing strategies 

• Effective learning and performance strategies 

• Knowledge, skills, and learning styles of experts 

b. Provide outputs from the data analysis, including a visual representation(s) of the 
knowledge structure. For example, present results in terms of: 

• Network scaling showing the interrelationship among items through a series 
of linked nodes. (See an example in Figure 1.) 

• Tree diagram 
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2.6 EsSUUa. (Continued) 

• Mental model (See an example in Figure 2.) 

• Schema 

b. Provide outputs from the data analysis, including a visual representation (s) of the 
knowledge structure. (Continued) 

• Hierarchical cluster depiction (showing groupings of related items within a 
hierarchy) 

• Multidimensional scaling depiction (showing how items interrelate along 
key psychological dimensions or axes) 

• Systematic grammar networks (networks showing how knowledge elements 
are categorized and the levels at which category distinctions are made) 

• Repertory grid (grid showing the important psychological distinctions 
between concepts) 

c. Discuss the conceptual and procedural knowledge needed for the task as a whole, 
including key domain concepts and hov; they interrelate, and important task 
components or subgoals. 

d. Identify strategies, heuristics, algorithms, or aids used for learning or in job 
performance, including: 

• Production rules 

• IF.... THEN rules 

• Heuristics 

• GOMS (Goals, Operators, Means, Selection Rules) 

• Decision trees 

• Rules of thumb 

• Effective job aids 

e. If automaticity or time-sharing: skills are one focus of the analysis, identify task 
components that are consistent. Note that consistent task components often do 
not correspond to task elements and activities. 
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2.6 Results . (Continued) 

f. Discuss the differences between the subject groups, including: 

• Similarities between groups 

• Differences between groups, and reasons for the differences 

• Learning progression and skill acquisition 

• Learner characteristics 

g. Discuss the interrelationships among results obtained and the models developed. 
For example, how does the expert-typical knowledge structure support the 
learning or execution of performance strategies? How does it support problem- 
solving? 

2.7 Implications of Results for the Design Phase . Discuss what the results and the models 
mean for training design, covering the following points: 

a. Recommended Training Structure. Discuss the content and organization of 
instructional material with reference to the knowledge organization. Discuss how 
best to present instructional material initially and how to facilitate progression 
to effective knowledge representations: 

• Using the expert model as an organizational framework both for learning 
and for job performance 

• Using the expert model to diagnose and remedy novice errors 

• Improving organization to support problem-solving 

• Improving understanding of interrelationships 
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2.7 Tmplications of Results for the Design Phase. (Continued^ 

b. Recommended Training Sequence. Discuss sequencing of instruction according 
to skill, knowledge, and knowledge structure progressions. The discussion 
should cover: 

• Job tasks 

• Task subgoals 

• Knowledge domains 

• Mental model components 

• Cognitive skills (e.g., teaching attention allocation before advanced 
problem-solving) 

• Strategies (e.g., simpler strategies before more complex) 

c. Recommended Instructional Setting, Media, and Strategies. Discuss the 
conditions that best facilitate learning according to the types of knowledge, skills, 
and knowledge structures underlying expertise and skill acquisition. For 
example: 

• Types of learning methods and media 

• Necessity of simulators for cognitive skill acquisition and practice 

• Use of mnemonics and organizational aids to facilitate memorization 

• Need to provide model-building learning activities to facilitate mental model 
development 

• Need to provide different training modes to accommodate individual 
differences in learning styles 
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Network representation of the average cognitive structure of naive programmers 




Network representation of the average cognitive structure of expert programmers 



HGURE 1. EXAMPLES OF NETWORK SCALING 
(COOKE & MCDONALD, 1987) 

From : Cooke, N. M., & McDonald, J. E. (1987). The application of psychological scaling techniques to 
knowledge elicitation for knowledge-based systems. International Journal of Man-Machine Studies. 26, 
533-550. 
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FIGURE 2. EXAMPLE OF AN EXPERT MENTAL MODEL 

(REDDING ET AL., 1991) 

From ; Redding, R. E., Cannon, J. R., Liennan, B. C, Ryder, J. M., Purcell, J. A., & Seamster, T. L. 

(1991). The analysis of expert performance in the redesign of the en route air traffic control 
curriculum. Proceedings of the Human Factors Society > 35th Annual Meeting (pp. 1403-1407). Santa 
Monicat CA: Human Factors Society. 
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